Criterion-referenced Standards Research Articles

Implementation of standard setting for high-stakes objective structured clinical examinations

Background and purposeTo describe one institution's approach to transformation of high-stakes objective structure clinical examinations (OSCEs) from norm-referenced to criterion-referenced standards setting and to evaluate the impact of these changes on OSCE performance and pass rates. Educational activity and settingThe OSCE writing team at the college selected a modified Angoff method appropriate for high-stakes assessments to replace the two standard deviation method previously used. Each member of the OSCE writing team independently reviewed the analytical checklist and calculated a passing score for active stations on OSCEs. Then the group met to determine a final pass score for each station. The team also determined critical cut points for each station, when indicated. After administration of the OSCEs, scores, pass rates, and need for remediation were compared to the previous norm-referenced method. Descriptive statistics were used to summarize the data. FindingsOSCE scores remained relatively unchanged when switched to a criterion-referenced method, but the number of remediators increased up to 2.6 fold. In the first year, the average score increased from 86.8% to 91.7% while the remediation rate increased from 2.8% to 7.4%. In the third year, the average increased from 90.9% to 92% while the remediation rate increased from 6% to 15.6%. Likewise, the fourth-year average increased from 84.9% to 87.5% while the remediation rate increased from 4.4% to 9%. SummaryTransition to a modified Angoff method did not impact average OSCE score but did increase the number of remediations.

Establishing performance standards for child development: learnings from the ECDI2030

BackgroundStandards of early childhood development (ECD) are needed to determine whether children living in different contexts are developmentally on track. The Early Childhood Development Index 2030 (ECDI2030) is a population-level measure intended to be used in household surveys to collect globally comparable data on one of the indicators chosen to monitor progress toward target 4.2 of the Sustainable Development Goals: The proportion of children aged 24–59 months who are developmentally on track in health, learning and psychosocial well-being.MethodsTo define performance cut-scores for the ECDI2030 we followed a criterion-referenced standard setting exercise using the modified Angoff method. The exercise gauged the expectations from 15 global experts in ECD and was informed by representative population data collected in Mexico and the State of Palestine. The final calibrated age-specific performance cut-scores were applied to these data to estimate the proportion of children developmentally on track, disaggregated by background characteristics, including the child's sex and attendance to early childhood education.ResultsThrough a process of standard setting, we generated robust performance standards for the ECDI2030 by establishing five age-specific cut-scores to identify children as developmentally on track.ConclusionsThis paper demonstrated how the standard setting methodology, typically applied to measures in the health and education fields, could be applied to a measure of child development. By creating robust criterion-referenced standards, we have been able to ensure that the cut-scores related to age for the ECDI2030 are based on performance standards set by global experts in the ECD field for defining on and off track development.

Criterion-Referenced Standards of Handgrip Strength for Identifying the Presence of Hypertension in Croatian Older Adults.

It is well known that muscular fitness has been associated with hypertension. However, it is less known which cut-off values of muscular fitness may predict the presence of hypertension. The main purpose of this study was to establish criterion-referenced standards of muscular fitness to define the presence of hypertension in Croatian older adults. In this cross-sectional study, we recruited men and women over 60 years of age. Muscular fitness was assessed by handgrip strength and normalized by height squared. Hypertension was defined as having systolic blood pressure ≥130 mm/Hg or diastolic blood pressure ≥80 mm/Hg. In older men, the optimal cut-point of muscular fitness in defining hypertension was 15.4 kg/m2. The area under the curve (AUC) was 0.85 (96% CI 0.77 to 0.92, p < 0.001). In older women, the optimal cut-point was 11.8 kg/m2, with an AUC of 0.84 (95% CI 0.80 to 0.89, p < 0.001). Men and women with cut-points of < 15.4 kg/m2 and < 11.8 kg/m2 were 11.8 (OR = 11.8, 95% CI 4.3 to 32.4, p < 0.001) and 10.6 (OR = 10.6, 95% CI 5.7 to 19.7, p < 0.001) times more likely to be diagnosed with hypertension. Our newly developed cut-points of muscular fitness assessed by the handgrip strength and normalized by height squared have satisfactory predictive validity properties in detecting men and women aged 60-81 years with hypertension.

Growth and Utility of Language and Early Literacy Measures for Young Children

Ongoing expansion of early education services in the United States is often rooted in these programs’ contribution to development that promotes later academic and behavioral competence. Multi-tiered systems of support (MTSS, sometimes referred to as Response to Intervention) represent one increasingly common resource to promote desired outcomes for all children. MTSS requires seasonal screening of all students and progress monitoring of those selected for more intensive intervention. This study explores development and evaluation of measures for these uses, designed specifically to assess the language and early literacy development of 3-year-old children. Results from assessment of 449 children describe measure reliability, growth across three seasonal screenings, development of criterion-referenced benchmarks, and variations in growth for children at different levels of initial performance. Results are discussed in terms of normative versus criterion-referenced standards for building MTSS assessment systems as well as future directions for research, policy, and practice.

Characterizing individual trainee learning curves in surgical training: Challenges and opportunities

The surgical learning curve is an observable and measurable phenomenon. In the era of competency-based approaches to surgical training, monitoring the trajectory of individual trainee competence attainment could represent a meaningful method of formative and summative assessment. While technology can assist this approach, a number of significant barriers to the implementation of such assessment methods remain, including: accurate data collection, standard setting, and reliable assessment. Translating individual learning curve data into quantifiable case minimum targets in training poses further difficulties, and may not be possible for all procedures, particularly those that are less frequently performed and assessed. In spite of these challenges, significant benefits could be realized through an individualized approach to competency assessment using trainee learning curve data. Tracking competence acquisition against criterion-referenced standards could allow for targeted training and remediation, conforming with modern theories of adult education and empowering trainees to take control of their own learning. Learning curve data could also be used to assess the effects of educational interventions such as simulation-based training on subsequent competence acquisition rates. Ultimately, the individual learning curves of trainees could be used to inform personalised decisions regarding entrustment, credentialing, and certification, allowing training programmes to move beyond minimum operative experience targets as a crude proxy measure of competence.

Collaboration readiness: Developing standards for interprofessional formative assessment.

Graduate nursing education strives to promote collaborative practice on interprofessional teams. However, measuring collaboration during formative stages of professional development is rare. Few assessments are available to determine whether graduates of nursing education programs have met the required accreditation benchmarks. This project evaluates two performance standards for assessing collaboration during formative assessments. Formative assessment of communication and collaboration skills was explored for 62 entry-level Master's students in nursing, using a team-based objective structured clinical examination (OSCE) involving two patient simulations and assessment checklists, and comparing normative (relative standard) and criterion-referenced (absolute standard) methods. Mean percentages of behaviors using the normative approach indicated a range of performance levels for particular communication and collaboration behaviors. Criterion-referenced standards were higher than normative standards from actual student performance. This method improves assessment of collaboration through OSCEs using normative and criterion-referenced standards. We recommend using both methods jointly when setting standards or qualifying scores.

Refining the FitnessGram with criterion-referenced Standards for Musculoskeletal Fitness

Refining the FitnessGram with criterion-referenced Standards for Musculoskeletal Fitness

Development of Criterion-Referenced Standards for Musculoskeletal Fitness in Youth: Considerations and Approaches by the FitnessGram Scientific Advisory Board

ABSTRACT The FitnessGram program has prioritized the use of evidence-based, criterion-referenced standards to advance health-related fitness assessments in physical education. The Institute of Medicine (IOM)1highlighted the importance of strength and power for youth fitness assessment and surveillance. The report specifically recommended the use of handgrip and long-jump, but a barrier to adoption was the lack of criterion-referenced standards to categorize and evaluate performance. The purpose of this paper is to provide an overview of a multi-faceted research initiative focused on advancing research on musculoskeletal fitness assessments in youth. The report chronicles the concepts, approaches, and considerations that shaped the plans for this large-scale research effort aimed at developing robust criterion-referenced fitness standards for these assessments.

The association between student body mass index and tests of flexibility assessed by the FITNESSGRAM®: New York City public school students, 2017-18.

FITNESSGRAM® is the most widely used criterion-referenced tool to assess/report on student health-related fitness across the US. Potential weight-related biases with the two most common tests of musculoskeletal fitness–the trunk extension and Back-Saver Sit-and-Reach (sit-and-reach)—have been hypothesized, though have not been studied. To determine the association between musculoskeletal fitness test performance and weight status, we use data from 571,133 New York City public school 4th-12th grade students (85% non-White; 75% qualified for free or reduced-price meals) with valid/complete 2017–18 FITNESSGRAM® data. Adjusted logistic mixed effects models with a random effect for school examined the association between weight status and whether a student was in the Healthy Fitness Zone (HFZ; met sex- and age-specific criterion-referenced standards) for the trunk extension and sit-and-reach. Compared to students with normal weight, the odds of being in the HFZ for trunk extension were lower for students with underweight (OR = 0.77; 95% CI: 0.741, 0.795) and higher for students with overweight (OR = 1.10; 95% CI: 1.081, 1.122) and obesity (OR = 1.11; 95% CI: 1.090, 1.13). The odds of being in the HFZ for sit-and-reach were lower for students with underweight OR = 0.85; 95% CI: 0.826, 0.878), overweight (OR = 0.83; 95% CI: 0.819, 0.844) and obesity (OR = 0.65; 95% CI: 0.641, 0.661). Students with overweight and obesity perform better on the trunk extension, yet worse on the sit-and-reach, compared to students with normal weight. Teachers, administrators, and researchers should be aware of the relationship of BMI with student performance in these assessments.

Design and Comparison of Criterion-referenced Standards for Grip Strength in U.S. Children and Adolescents

ABSTRACT We designed grip strength standards for youth by back-extrapolating cutpoints based on weakness at age 65. Existing standards, based on insufficient bone mineral content (BMC), were also compared in a nationally representative sample. Altogether, 6837 participants (n = 2199 ≤ 18.9 years of age) from the National Health and Nutrition Examination Survey had dominant hand, grip strength assessed. Lambda Mu Sigma regression was used to create percentiles of strength and identify thresholds linked with clinically relevant weakness (CRW) in older adults. Approximately 93% and 75% of young people met standards based on CRW and BMC, respectively. Non-Hispanic white and black youth were more likely to meet grip strength cutpoints. Underweight children were less likely (odds ratios = 0.20 to 0.26), while overweight/obese youth were more likely (1.38 to 2.71), to meet strength standards compared to normal weight. The standards can be used with, or in place of, normative standards when evaluating grip strength.

Health-Related Motor Testing of Children in Primary School: A Systematic Review of Criterion-Referenced Standards.

Being physically fit in younger years prevents several diseases in the presence as well as in the life course. Therefore, monitoring physical fitness and motor competence through motor testing is essential for determining developmental status and identifying health-related risks. The main objectives of this systematic review were (1) to identify currently available health-related criterion-referenced standards and cut-off points for physical fitness and motor competence test items, (2) to frame the methodological background on setting health-related criterion-referenced standards and (3) to give implications for a health-related evaluation system for physical fitness and motor competence tests. The electronic data base search (PubMed, Web of Science and SURF) yielded 2062 records in total and identified six empirical studies reporting cut-off points of motor test items for children (7–10 years), as well as 30 methodological papers discussing determination approaches to health-related criterion-referenced standards. Data collection, selection and analyses followed the PRISMA guidelines. Health-related motor test standards need to be gender- and age-specific but should refer to an absolute cut-off point rather than to relative performance in the reference group. Due to the lack of data on health-related criterion referenced standards, receiver-operating-characteristic (ROC) curves provide a tool for the determination of cut-off points and criterion referenced standards for physical fitness and motor competence tests. A standardized approach forms the fundamental base for a globally applicable evaluation of health-related fitness tests.

AGREEMENT BETWEEN PHYSICAL BEST AND FITNESSGRAM CRITERION-REFERENCED STANDARDS FOR MUSCULAR STRENGTH AND ENDURANCE.

ABSTRACTObjective:To verify the agreement between PHYSICAL BEST and FITNESSGRAM health-related criteria for muscle strength and endurance.Methods:This agreement study had a sample of 81 children and adolescents. Participants were submitted to the PHYSICAL BEST (Sit-up and Pull-up) and FITNESSGRAM (Curl-up and Modified Pull-up) test batteries. Additionally, FITNESSGRAM also proposed criteria for Pull-up test. Results of tests were classified in accordance with their respective criteria. Each group had an interval of seven days between the first and second battery of tests. Statistical analysis used the Kappa index (p<0.05).Results:Sit-up and Curl-up tests among the boys agreed in 72.2% (Kappa=0.368; p=0.004) of cases, and for the girls, in 64.4% (Kappa=0.130; p=0.076). Pull-up (PHYSICAL BEST versus FITNESSGRAM) agreed in 83.3% (Kappa=0.599; p<0.001) for boys. The agreement between Pull-up and Modified Pull-up (PHYSICAL BEST versus FITNESSGRAM) for boys was 47.2% (Kappa=0.071; p=0.533), and for girls, 44.5% (Kappa=0.102; p=0.120). The agreement between the Pull-up and Modified Pull-up tests (FITNESSGRAM) for boys was 58.4% (Kappa=0.215; p=0.143), and for girls, 44.5% (Kappa=0.102; p=0.120).Conclusions:For individual analysis over time, as well as for the comparison of passing rates between different populations, caution is advised when using different criterion-referenced standards for strength and endurance, particularly if using different tests.

중고등부 골프선수의 체력 및 비거리 검사를 위한 준거지향 기준 설정

중고등부 골프선수의 체력 및 비거리 검사를 위한 준거지향 기준 설정

Evidence-Based Optimal Cutoff Values with the Validation of Criterion-Referenced Standards for Sarcopenic Elderly Fitness Improvement.

This study provides a newly updated perspective of information on severely screened 21 previous studies of the various measurement methods for improving physical fitness and providing determined cutoff values from our reserved elderly human database. We aimed to provide scientific evidence-based information regarding physical fitness standards for developing useful prognostics, promoting and maintaining health programs for sarcopenic elderly. 21 previous studies emphasizing criterion referenced standards and receiver operator characteristic (ROC) curve analyses for improving physical fitness were screened. For predicting the prevalence of sarcopenia, the t-test, logistic regression, linear regression, ROC curve analyses, and voluntary categorizations such as the twentieth or sixtieth percentile classification were used. Based on these scientific evidences, we determined cutoff values from our reserved DB and realized that 75 years for men and 70 years for women are the transitional period during which there are large declines in muscle and fat mass (p < 0.01), which reflects physical function tests (p < 0.01) in both genders. Using the six factors with ideal cutoff thresholds, an individual exercise program can be designed for alleviating symptoms of frailty caused by sarcopenia for the elderly.

Comparison of blood pressure across body composition and cardiorespiratory fitness zones determined in accordance with FITNESSGRAM® criterion-referenced standards

PurposeTo compare systolic and diastolic blood pressures using the FitnessGram criterion-referenced standards for body composition and cardiorespiratory fitness in Brazilian young people.MethodsThe study had a cross-sectional design. With institutional approval, a sample of 874 students aged 13–18 years enrolled in public schools in a medium size town in Southern Brazil took part in this study. Body mass index was calculated to represent the subjects’ body composition. The progressive aerobic cardiovascular endurance run (PACER) test was applied to estimate aerobic capacity. The participants were classified by using the FitnessGram criterion-referenced standards. Systolic and diastolic blood pressures were assessed with the oscillometric method.ResultsMultivariate analyses of covariance indicated that the blood pressure of boys (Wilks’ lambda = 0.97, F (6, 834) = 2.48, &lt;i&gt;p&lt;/i&gt; &lt; 0.05, ES = 0.02) and girls (Wilks’ lambda = 0.94, F (6, 886) = 4.89, &lt;i&gt;p&lt;/i&gt; &lt; 0.01, ES = 0.03) differed between fitness zones for body composition. The analyses showed that blood pressure also varied across cardiorespiratory fitness zones for girls (Wilks’ lambda = 0.98, F (4, 886) = 2.69,&lt;i&gt; p&lt;/i&gt; &lt; 0.05, ES = 0.01) but not for boys.ConclusionsReaching the health fitness zone of the criterion-referenced standards for body composition seems to be associated with lower blood pressure in boys and girls. Cardiorespiratory fitness had a similar effect solely in girls.

Exploring differences in individual and group judgements in standard setting.

Standard setting is critically important to assessment decisions in medical education. Recent research has demonstrated variations between medical schools in the standards set for shared items. Despite the centrality of judgement to criterion-referenced standard setting methods, little is known about the individual or group processes that underpin them. This study aimed to explore the operation and interaction of these processes in order to illuminate potential sources of variability. Using qualitative research, we purposively sampled across UK medical schools that set a low, medium or high standard on nationally shared items, collecting data by observation of graduation-level standard-setting meetings and semi-structured interviews with standard-setting judges. Data were analysed using thematic analysis based on the principles of grounded theory. Standard setting occurred through the complex interaction of institutional context, judges' individual perspectives and group interactions. Schools' procedures, panel members and atmosphere produced unique contexts. Individual judges formed varied understandings of the clinical and technical features of each question, relating these to their differing (sometimes contradictory) conceptions of minimally competent students, by balancing information and making suppositions. Conceptions of minimal competence variously comprised: limited attendance; limited knowledge; poor knowledge application; emotional responses to questions; 'test-savviness', or a strategic focus on safety. Judges experienced tensions trying to situate these abstract conceptions in reality, revealing uncertainty. Groups constructively revised scores through debate, sharing information and often constructing detailed clinical representations of cases. Groups frequently displayed conformity, illustrating a belief that outlying judges were likely to be incorrect. Less frequently, judges resisted change, using emphatic language, bargaining or, rarely, 'polarisation' to influence colleagues. Despite careful conduct through well-established procedures, standard setting is judgementally complex and involves uncertainty. Understanding whether or how these varied processes produce the previously observed variations in outcomes may offer routes to enhance equivalence of criterion-referenced standards.

Developing Criterion-Referenced Standards for Cardiorespiratory Fitness Among Canadian Adults Aged 18-69 Years

PURPOSE: Cardiorespiratory fitness levels among adults are an important indication of general health. The aim of this project was to develop criterion-referenced standards for estimates of cardiorespiratory fitness in Canadian adults aged 18-69 years using a nationally representative sample, and body mass index (BMI) as the criterion measure. METHODS: Cross-sectional data were obtained from cycles 1 (2007-09) and 2 (2009-11) of the Canadian Health Measures Survey. The modified Canadian Aerobic Fitness Test (mCAFT) was used to predict cardiorespiratory fitness (as VO2max). Measured height and weight were used to calculate BMI. Obesity was classified as having a BMI greater than 30 kg•m-2. Receiver operating characteristic curves were used to identify cardiorespiratory fitness standards that optimized both sensitivity and 1-specificity (Youden’s J) for males and females, separately, across five age groups (i.e., 18-29, 30-39, 40-49, 50-59, and 60-69 years). RESULTS: A total of 4,967 participants (53% female, Mage=41.0±13.8 years) were retained for the present analyses. The area under the curve ranged from 0.79-0.86 in males, and 0.85-0.92 in females. The optimal standards for cardiorespiratory fitness using BMI as the criterion measure ranged from 13.1-31.7 mL•kg-1•min-1 for males, and 16.9-32.6 mL•kg-1•min-1 for females. The cardiorespiratory fitness standards were slightly higher for females in comparison with males when matched for age. CONCLUSIONS: These cardiorespiratory fitness standards may be useful in public health settings, particularly in Canada. Future research should further investigate other criterion measures, and research should validate these cut-points to see if they adequately discriminate individuals with chronic diseases.

Health-related physical fitness and weight status in 13- to 15-year-old Latino adolescents. A pooled analysis

ObjectiveThe aim of this study was to investigate the relation between health-related physical fitness and weight status in 13- to 15-year-old Latino adolescents. MethodThe final sample consisted of 73,561 adolescents aged 13–15 years (35,175 girls) from Chile (n=48,771) and Colombia (n=24,790). Cardiorespiratory and musculoskeletal fitness were measured using 20-m shuttle run (relative peak oxygen uptake – VO2peak) and standing broad jump test (lower body explosive strength), respectively. The International Obesity Task Force definition was used to define weight status (i.e., underweight, normal weight, overweight, and obese). ResultsThe present study found an inverted J-shape relationship between body mass index, cardiorespiratory fitness, and musculoskeletal fitness in both genders and all age groups (p<0.01). Results also suggest that underweight adolescents, and not just overweight and obese adolescents, have lower odds of having a healthy cardiorespiratory fitness (based on new international criterion-referenced standards) profile when compared with their normal weight peers, except in girls aged 14 (p=0.268) and 15 years (p=0.280). ConclusionsThe present results indicate low cardiorespiratory fitness and musculoskeletal fitness levels in underweight, overweight, and obese adolescents when compared with their normal weight peers. The findings appear to suggest that exercise programs should to decrease fat mass in overweight/obese adolescents and increase muscle mass in underweight adolescents.

Exploring the utility of cardiorespiratory fitness as a population health surveillance indicator for children and youth: An international analysis of results from the 20-m shuttle run test.

Emerging evidence has demonstrated the strong link between cardiorespiratory fitness and multiple aspects of health (i.e., physiological, physical, psychosocial, cognitive), independent of physical activity, among school-aged children and youth. Cardiorespiratory fitness is a trait that does not vary substantially from day-to-day, and provides an indication of recent physical activity levels, making it an important possible indicator of population health. Thus, the objective of this dissertation was to investigate the utility of cardiorespiratory fitness, measured using the 20-m shuttle run test, as a broad, holistic health indicator for population health surveillance among children and youth. To achieve this objective we completed 7 manuscripts, all prepared for submission to peer-reviewed, scientific journals: (1) Systematic review of the relationship between 20-m shuttle run performance and health indicators among children and youth. (2) Review of criterion-referenced standards for cardiorespiratory fitness: what percentage of 1 142 026 international children and youth are apparently healthy? (3) Cardiorespiratory fitness is associated with physical literacy in a large sample of Canadian children aged 8 to 12 years. (4) International variability in 20-m shuttle run performance in children and youth: Who are the fittest from a 50-country comparison? A systematic review with pooling of aggregate results. (5) Making a case for cardiorespiratory fitness surveillance among children and youth. (6) International normative 20-m shuttle run values from 1 142 026 children and youth representing 50 countries. (7) Temporal trends in the cardiorespiratory fitness of children and adolescents representing 19 high-income and upper middle-income countries between 1981 and 2014. Combined, this dissertation provides support for the importance of cardiorespiratory fitness for health surveillance among school-aged children and youth. Results from the international analysis highlighted the variability across countries, with countries in north-central Europe and Africa having the highest cardiorespiratory fitness, and countries in South America having the lowest cardiorespiratory fitness. The results indicated that declines in cardiorespiratory fitness may have stabilized in recent years in some high- and middle-income countries. This dissertation also identified 2 methods (criterion- and normative-referenced standards) to interpret cardiorespiratory fitness levels among children and youth, methods that could be used to inform future consensus, surveillance, and cardiorespiratory fitness guidelines.

Criterion-Referenced Fitness Standards Associated with Maintaining Functional Capacity in Chilean Older Adults.

The aim of the current study was to develop fitness standards associated with maintaining physical capacity in older adults at two different levels-that needed for advanced functioning and that needed for maintaining moderate (independent) functional ability. This was a cross-sectional study of 406 Chilean adults aged 60 and more. Functional capacity was measured through the Composite Physical Function (CPF) scale. A battery of validated fitness tests for this population was used. Based on the results of the CPF, two variables were created: "advanced physical capacity" (defined as those scoring 24 on the CPF scale) and "moderate (independent) physical capacity," defined as those scoring between 14 and 23 on the CPF scale. Fitness cut-off values were calculated for each of the two created variables to evaluate the risk of losing the independence through Receiver Operating Characteristic (ROC) curves analysis and logistic regression. The developed cut-off points provide important information for professionals working directly with older adults, to detect the risk of losing functionality and independence.